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As  requested,  this  report  conveys  the  results  of  our  review  of  the  National 
Aeronautics  and  Space  Administration’s  (nasa)  performance  plan  for  jBscal 
year  1999.  Under  the  Government  Performance  and  Results  Act  of  1993, 
also  known  as  “the  Results  Act,”  nasa  was  required  to  submit  to  Congress 
its  first  performance  plan,  for  fiscal  year  1999,  along  with  the  President’s 
fiscal  year  1999  budget  request.  According  to  NASA  officials,  the  plan  was 
provided  to  Congress  on  March  27, 1998.  NASA  views  its  performance  plan 
as  the  third  component  of  its  “trilogy  of  Results  Act-supporting” 
documents,  the  other  two  being  the  annual  budget  justification  and  the 
strategic  plan. 

Our  report  discusses  (1)  NASA’s  goals  and  objectives,  including  how  the 
agency  plans  to  measure  its  progress  towards  achieving  these  goals  and 
objectives,  (2)  the  agency’s  strategies  and  resources  needed  to  achieve  its 
goals,  and  (3)  the  availability  and  rehability  of  data  necessary  to  achieve 
progress. 


Background 


The  Results  Act  requires  that  the  Office  of  Management  and  Budget  (omb) 
direct  each  executive  agency  to  prepare  an  annual  performance  plan, 
beginning  with  fiscal  year  1999,  The  performance  plan  is  one  of  three 
components  of  the  Results  Act,  the  others  being  the  strategic  plan, 
submitted  by  agencies  in  September  1997,  and  the  annual  report  due 
March  31, 2000.  The  performance  plan  should  describe  (1)  annual 
performance  goals  and  measures,  (2)  the  strategies  and  resources  to 
achieve  those  goals,  and  (3)  procedures  to  verify  and  validate  reported 
performance.  The  act  requires  that  performance  goals  be  linked  to  the 
program  activities  in  agencies’  budgets  and  be  expressed  in  an  objective, 
quantifiable,  and  measurable  form,  omb  is  to  use  agency  performance  plans 
to  develop  the  overall  federal  government  performance  plan  submitted 
annually  to  Congress,  beginning  with  the  President’s  fiscal  year  1999 
budget. 


Page  1 


GAO/NSIAD-98-181  NASA’s  Fiscal  Year  1999  Performance  Plan 


Results  in  Brief 


We  found  that  nasa’s  fiscal  1999  performance  plan  could  provide  a  clearer 
picture  of  intended  performance  across  the  agency,  does  not  fully  portray 
how  NASA’s  strategies  and  resources  will  help  it  achieve  the  plan’s 
performance  goals,  and  partially  provides  confidence  that  the  information 
NASA  wiU  use  to  assess  performance  wiU  be  accurate,  complete,  and 
credible. 

Among  its  strengths,  nasa’s  performance  plan  reflects  tbe  mission 
statement  and  goals  in  its  strategic  plan  and  provides  good  linkage 
between  these  strategic  goals  and  the  performance  plan’s  goals  and 
targets;  incorporates  performance  measures  that  are  generally  objective, 
quantifiable,  and  useful  for  assessing  progress  toward  the  plan’s 
performance  objectives;  and  provides  for  annual  external  assessments  by 
its  Advisory  Council  and  semi-annual  internal  assessments  by  its  Senior 
Management  Council  to  validate  progress  toward  meeting  the  agency’s 
goals  and  objectives. 

To  make  the  plan  more  useful  for  purposes  of  the  Results  Act,  nasa’s 
performance  plan  should  better  link  performance  goals  and  measures  to 
the  program  activities  in  nasa’s  budget;  more  fully  explain  nasa’s 
procedures  for  verifying  and  validating  performance  data  by  recognizing 
the  limitations  that  affect  the  credibility  of  data  that  wiU  be  used  to 
measure  performance;  and  acknowledge  nasa’s  major  management 
challenges  and  associated  corrective  actions  in  order  to  provide  a  more 
comprehensive  understanding  of  the  importance  of  the  goals  and 
performance  measures  chosen  for  its  internal  crosscutting  processes. 

Some  of  the  concerns  we  have  regarding  nasa’s  performance  plan  are 
similar  to  our  observations  on  nasa’s  strategic  plan  issued  on 
September  30, 1997.  For  example,  NASA’s  strategic  plan  did  not  contain 
evidence  that  NASA  had  coordinated  the  plan  with  those  agencies  whose 
programs  and  activities  complement  NASA’s  and  did  not  discuss  the  human, 
capital,  and  information  resources  needed  to  achieve  the  goals  and 
objectives  in  the  plan. 


NASA’s  Performance 
Plan  Could  Provide  a 
Clearer  Picture  of 
Intended  Performance 
Across  the  Agency 


Generally,  the  performance  goals  in  nasa’s  plan  are  objective  and  the 
performance  measures  are  quantifiable  and  useful  for  assessing  progress. 
While  the  plan  reflects  the  mission  statement  and  strategic  goals  in  nasa’s 
strategic  plan  and  provides  clear  Unks  between  strategic  goals  and  the 
annual  performance  goals  and  measures,  it  does  not  clearly  associate  the 
plan’s  performance  goals  and  measures  with  program  activities  in  the 
budget.  Although  nasa’s  plan  recognizes  that  oliier  agencies  and  partners 
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Defining  Expected 
Performance 


have  complementary  missions  and  activities  and  acknowledges  that  nasa 
must  work  closely  with  them  to  achieve  its  goals,  it  does  not  discuss  the 
extent  to  which  it  has  coordinated  either  its  strategies  for  achieving  goals 
or  its  development  of  performance  measures. 


The  performance  plan  provides  near-term  goals  for  each  of  nasa’s  fom 
business  enterprises  and  at  least  one  performance  target  to  address  each 
of  the  goals  for  fiscal  year  1999.  The  four  enterprises,  which  were 
characterized  by  nasa  in  its  fiscal  year  1999  strategic  plan  as  analogous  to 
business  units  in  the  private  sector,  are  Space  Science;  Earth  Science; 
Human  Exploration  and  Development  of  Space;  and  Aeronautics  and 
Space  Transportation  Technology.  In  addition,  the  performance  plan 
includes  goals  and  measures  for  the  internal  processes  it  characterizes  as 
crosscutting. 

NASA  officials  told  us  that  the  enterprise  performance  targets  are  indicators 
of  progress  toward  the  agency’s  near-term  goals.  Generally,  the  enterprise 
near-term  goals  are  objective  and  free  from  bias.  The  plan  provides  both 
quantitative  and  qualitative  measures  for  aspects  of  some  of  the  goals.  For 
example,  in  addition  to  quantitative  measirres  for  exploring  the  role  of 
gravity  in  physical,  chemical,  and  biological  processes,  the  plan  provides 
for  using  “the  data  obtained  by  fluid  physics  experiments  on  suspensions 
of  colloidal  particles  on  MSL-1  [Microgravity  Science  Laboratory]  to 

answer  fundamental  questions  in  condensed  matter  physics ”  nasa 

officials  acknowledged  that  this  would  require  the  use  of  qualitative 
measures  to  determine  whether  the  fundamental  questions  have  been 
answered. 

OMB  recortunends  that  outcome  goals  be  included  in  a  performance  plan 
whenever  possible,  but  recognizes  that  agencies  will  supplement  outcome 
goals  with  output  goals.  Some  of  NASA’s  performance  measures  are 
outcome  oriented;  others  are  output  oriented.  NASA  officials  told  us  that 
outputs  are  used  when  demonstrating  intermediate  progress  in  support  of 
future  outcomes  and  often  reflect  program  commitments  made  in 
conjunction  with  science  users.  For  instance,  they  said  that  the 
performance  target  of  orbiting  the  asteroid  Eros  closer  than  50 
kilometers — a  performance  measure  associated  with  nasa’s  objective  of 
exploring  the  solar  system — ^matches  Near-Earth  Rendezvous  program 
expectations  and  specifications.  A  few  performance  goals  are  not  specific 
to  fiscal  year  1999;  rather,  they  span  multiple  years.  For  example,  the  plan 
provides  for  reducing  “average  spacecraft  cost  for  Space  Science  and 
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Earth  Science  missions  to  $190  million  from  $590  million.”  However,  the 
performance  target  is  based  on  comparing  spacecraft  programs  completed 
in  the  time  period  of  fiscal  years  1990  through  1994  with  those  completed 
between  fiscal  years  1995  through  1999.  NASA  officials  told  us  that 
expressing  multiple  year  programs  in  a  single-year  context  required  by  the 
Results  Act  was  difficult  and  a  source  of  fliistration  for  the  agency. 

In  order  to  provide  a  more  comprehensive  understanding  of  the 
importance  of  the  goals  and  performance  measures  chosen  for  its 
crosscutting  processes,  it  would  have  been  helpful  for  the  plan  to 
acknowledge  the  agency’s  major  management  challenges  and  associated 
corrective  actions.  For  example,  nasa  officials  told  us  that  the  plan’s  target 
of  achieving  70  percent  or  greater  of  resource  authority  obligated  relates 
to  funding  carryovers — an  issue  we  have  identified  as  a  major  challenge^ 
and  which  the  NASA  Administrator  has  recognized  in  congressional 
testimony  as  a  top  priority.  We  have  also  identified  other  management 
challenges.  These  include: 

•  improving  systems  and  mformation  for  monitoring  contractor  activities 
and  complying  with  contract  management  requirements; 

•  implementing  a  new  accoimting  system  and  the  importance  of  this  effort 
to  providing  Congress  and  agency  management  with  better  financial 
information  and  allowing  the  agency  to  move  to  full  cost  accounting;  and 

•  resolving  imeven  progress  toward  better  NASA  and  Department  of  Defense 
cooperation  and  developing  a  national  perspective  on  aerospace  test 
facilities. 

OMB  guidance  states  that  performance  goals  for  corrective  steps  for  major 
management  problems  should  be  included  for  problems  whose  resolution 
is  mission-critical,  such  as  problems  that  could  materially  impede  the 
achievement  of  program  goals.  Recogmzing  those  management  challenges 
appears  to  meet  the  intent  of  omb’s  guidance. 

Furthermore,  although  the  performance  plan  includes  performance  targets 
related  to  contracting,  such  as  increasing  the  obligated  funds  for 
Performance-Based  Contracts  to  80  percent,  the  performance  measme  can 
be  better  understood  by  referencing  the  agency’s  efforts  to  ensure  that  it 
has  relevant  and  reliable  methods  for  timely  and  accurate  monitoring  of  its 
contract  management  activities.  For  example,  the  performance  plan  is 


^NASA:  M^or  Management  Problems  (GAQ/l-NSIAD-97-178,  July  24, 1997). 
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Connecting  Goals, 
Missions,  and  Activities 


Recognizing  Crosscutting 
Efforts 


silent  on  NASA’s  planned  corrective  efforts  in  fiscal  year  1999  to  evaluate 
the  effectiveness  of  its  contract  management  activities  at  its  field  centers.^ 


NASA’s  performance  goals  are  closely  aligned  with  the  agency-wide  and 
enterprise  strategic  mission,  goals,  and  objectives,  and  provide  good 
linkages  between  the  agency’s  near-term  strategic  goals  and  the  enterprise 
near-term  goals,  objectives,  and  performance  targets.  This  is  accomplished 
through  well-laid  out  charts  summarizing  these  relationships. 

OMB  Circular  A-11  provides  that  the  annual  performance  plan  should  also 
show  how  specific  performance  goals  are  related  to  the  specific  program 
activities  contained  in  the  agency’s  program  and  financing  (P&F) 
schedules  in  the  President’s  Budget,  nasa’s  performance  plan  contains  a 
table  that  allocates  nasa’s  $13.5  billion  fiscal  year  1999  budget  request 
among  its  four  enterprises  and  several  other  consolidated  budget 
accounts.  Each  of  the  enterprises  is  related  to  a  set  of  performance  goals. 
However,  the  plan  does  not  clearly  explain  how  nasa’s  program  activities 
relate  to  the  budget  totals  presented  for  each  enterprise.  There  is  no 
crosswalk  to  facilitate  an  understanding  of  the  relationship  between  the 
specific  program  activities  in  the  budget  and  the  goals  and  measures  in  the 
performance  plan.  Therefore,  we  believe  nasa’s  performance  plan  cannot 
help  Congress  rmderstand  which  performance  goals  and  measures  in  the 
plan  cover  which  program  activities  or  whether  all  program  activities  in 
NASA’s  budget  are  covered  by  its  performance  goals. 

NASA  officials  with  whom  we  spoke  during  our  review  indicated  that  these 
relationships  can  be  determined  by  simultaneously  analyzing  information 
in  its  “trilogy”  of  documents,  namely  the.annual  budget  justification, 
strategic  plan,  and  performance  plan.  However,  they  recognize  that  a 
clearer  connection  between  performance  goals  and  the  program  activities 
in  the  budget  needs  to  be  provided  in  future  performance  plans  to 
facilitate  analysis.  According  to  those  officials,  an  initiative  is  underway  to 
align  these  documents. 


'The  performance  plan  points  out  that  collaborative  efforts  with  other 
agencies  and  international  partners  will  continue  to  be  key  to  the 
successful  implementation  of  enterprise  strategies.  Although  the  plan 
identifies  the  specific  agency  or  international  partner  involved  in  carrying 
out  specific  efforts,  it  does  not  discuss  the  extent  to  which  other  agencies 


^High  Risk  Program:  Information  on  Selected  High-Risk  Areas  (GAO/HR-97-30,  May  16, 1997). 
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have  been  coordinated  with  in  establishing  the  goals,  objectives,  and 
associated  performance  targets.  For  example,  in  describing  the  objective 
of  developing  next-generation  computational  design  tools,  the  plan 
indicates  that  nasa’s  efforts  are  part  of  the  Federal  High  Performance 
Computing  and  Communications  (hpcc)  initiative.  However,  there  is  no 
discussion  as  to  whether  nasa  coordinated  its  performance  target  of  a 
200-fold  improvement  with  other  federal  partners,  nor  is  there  an 
explanation  of  how  nasa’s  effort  will  contribute  to  the  overall  federal 
initiative. 

NASA’s  performance  plan  would  be  strengthened  if  it  identified  activities 
being  undertaken  to  address  crosscutting  issues  of  broad  national  concern 
and  discussed  how  achieving  the  agency’s  goals  will  contribute  to 
addressing  the  crosscutting  issues.  To  indicate  how  nasa’s  programs 
contribute  to  addressing  a  crosscutting  issue,  the  plan  could  present 
output  goals,  output  measures,  intermediate  outcome  goals,  and 
intermediate  outcome  measures  that  reflect  the  agency’s  contributions  to 
addressing  the  crosscutting  issues. 

The  performance  plan  could  be  improved  if  it  discussed  efforts  to 
coordinate  NASA’s  programs  with  other  federal  programs  performing 
related  activities  and  provided  evidence  of  coordination  such  as  joint 
planning  and  coordination  or  development  of  partnerships  aimed  at 
improving  program  efficiency  and  ejffectiveness. 


NASA’s  Performance 
Plan  Does  Not  Fully 
Discuss  How  the 
Agency’s  Strategies 
and  Resources  Will 
Help  Achieve  Its 
Goals 

Connecting  Strategies  to 
Results 


The  performance  plan  does  not  fully  discuss  how  the  agency’s  strategies 
and  resources  wiU  help  achieve  its  goals.  The  implementation  strategy 
descriptions  in  each  enterprise  section  are  unclear  and  do  not  provide 
sufficient  information  on  how  NASA  will  go  about  accomplishing  its 
near-term  enterprise  goals.  The  plan  only  partially  discusses  resoiurces;  it 
does  not  describe  the  operational  processes,  skUl  and  technology,  human, 
capital,  and  information  resources  required  to  achieve  the  performance 
goals. 


The  NASA  plan  does  not  fully  describe  how  the  agency  will  achieve  its 
performance  goals,  nor  does  it  reference  other  documents  that  show  how 
the  strategies  will  contribute  to  achieving  the  agency’s  performance  goals. 
Enterprise  implementation  strategy  descriptions  lack  detailed  information 
on  how  NASA  will  accomplish  its  enterprise  goals.  In  several  instances, 
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these  descriptions  appear  to  outline  philosophies  rather  than  describe 
enabling  actions.  For  instance,  application  of  the  “faster,  better,  cheaper” 
approach  to  spacecraft  development  identified  in  both  the  Space  Science 
and  Earth  Science  implementation  strategy  sections  is  not  clearly 
explained.  At  a  minimum,  it  would  have  been  helpful  if  the  plan  had 
included  an  explicit  discussion  on  what  is  involved  in  applying  that 
approach  and  how  the  establishment  of  “prudent  risks”  is  part  of  it. 

While  not  required  by  the  Results  Act,  we  believe  that  a  similar  discussion 
of  external  factors  and  discussion  of  how  nasa  will  mitigate  or  use  the 
identified  factors  to  achieve  performance  goals  could  provide  additional 
context  regarding  anticipated  performance.  With  the  exception  of  stating 
that  NASA  assumes  that  International  Space  Station  partners  wlU  meet  their 
own  schedules,  the  plan  does  not  explain  how  NASA  will  address  external 
factors  that  could  affect  the  agency’s  performance  during  the  coming  fiscal 
year.  In  its  strategic  plan,  nasa  had  identified  several  key  external  factors, 
such  as  the  stability  of  future  budgets.  We  believe  that  a  similar  discussion 
in  the  performance  plan  would  be  helpful.  This  is  particularly  pertinent  for 
NASA’s  budgetary  programmatic  priorities,  including  the  International 
Space  Station,  which  could  potentially  consume  a  large  portion  of  nasa’s 
future  resources  and  affect  the  timely  implementation  of  other  programs. 


Connecting  Resources  to 
Strategies 


NASA’s  performance  plan  partially  discusses  the  resources  it  will  use  to 
achieve  the  performance  goals.  Characterization  of  resources  needed  by 
NASA  to  meet  its  performance  goals  is  limited  to  identifying  fiscal  year  1999 
funding  requirements  by  enterprise.  In  addition,  the  plan  does  not  discuss 
operational  processes,  skills  and  technology,  human  and  capital  resources 
required  to  meet  the  performance  goals. 

Information  technology  issues  are  discussed  in  the  context  of  the  internal 
crosscutting  objective  entitled  “improving  technology  capability  and 
services.”  'The  performance  target  is  identified  as  improving  information 
technology  return  on  investment  and  customer  satisfaction  by  maintaining 
a  positive  return  on  investment  and  a  “satisfactory”  rating  from 
information  technology  customers.  The  performance  measrrre  of 
“maintaining  a  positive  return  on  investment”  is  vague  and  does  not 
indicate  how  the  target  of  improving  information  technology  return  on 
investment  wUl  be  achieved.  Likewise,  although  maintaining  a 
“satisfactory”  rating  from  customers  is  a  positive  measure,  it  does  not 
indicate  whether  any  improvement  is  being  achieved. 
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We  observe  that  the  information  technology  goal  and  measiure  identified  in 
the  performance  plan  do  not  specifically  address  any  of  the  important 
information  technology  management  issues  that  NASA  cmrently  faces. 
Specifically,  the  plan  does  not  discuss  how 

•  NASA  plans  to  measure  compliance  with  the  Clinger-Cohen  Act  of  1996, 
which  calls  for  agencies  to  implement  a  framework  of  modem  technology 
management  based  on  practices  followed  by  leading  private-sector  and 
public-sector  organizations  that  have  successfully  used  technology  to 
dramatically  improve  performance  and  meet  strategic  goals; 

•  the  agency’s  new  Chief  Information  Officer  (cio)  management  structure, 
which  depends  on  the  cooperation  of  NASA’s  diverse  enterprises  and  field 
centers,  wiU  measure  its  success  in  providing  strategic  direction  to 
information  technology  investments;  and 

•  NASA  plans  to  address  the  “year  2000  problem”  (which  requires  that 
computer  systems  be  changed  to  accommodate  dates  beyond  the  year 
1999)  as  well  as  any  significant  information  secmity  weaknesses — ^two 
issues  that  we  have  identified  as  high  risk  across  the  federal  government.® 
Both  the  “year  2000  problem”  and  information  security  weaknesses  could 
seriously  rmdermine  the  performance  and  integrity  of  nasa’s  information 
systems  if  not  adequately  addressed. 

In  addition,  NASA’s  performance  plan  contains  no  discussion  of  its  strategy 
for  achieving  improvements  in  information  technology  capability  and 
services,  nor  does  it  refer  to  a  separate  information  technology  annual 
plan  that  might  contain  such  information.  As  a  result,  there  is  no  way  to 
assess  whether  NASA  is  reasonably  prepared  to  undertake  any  significant 
improvement  in  information  technology  capability  and  services  or  whether 
such  a  goal  is  attainable.  Furthermore,  the  performance  plan  gives  no 
indication  of  the  resources  associated  with  the  information  technology 
improvement  goal. 

Furthermore,  the  performance  of  NASA’s  major  capital  investments  in 
information  technology  are  not  specifically  addressed  in  its  performance 
plan.  For  example,  the  Earth  Science  Enterprise  section  discusses  an 
objective  of  improving  dissemination  of  Earth  Science  research  results  but 
does  not  mention  the  Earth  Observing  System  Data  and  Information 
System  (eosdis).  nasa  has  requested  $257  million  for  eosdis  in  fiscal  year 
1999.  No  planned  measures  of  the  performance  of  eosdis  are  included. 


year  2000  problem  refers  to  the  potential  for  computer  programs  to  generate  incorrect  results 
when  using  dates  from  the  year  2000  and  beyond. 
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NASA’s  Performance 
Plan  Partially 
Provides  Confidence 
That  Its  Performance 
Information  Will  Be 
Credible 


While  NASA’s  performance  plan  identifies  internal  and  external 
organizations  that  wiU  evaluate  performance,  it  does  not  include  a 
discussion  of  the  procedures  that  will  be  used  to  verify  and  validate 
performance  data  and  the  limitations  that  could  exist  with  internal  sources 
of  data  that  nasa  plans  to  use  to  assess  performance.  Although  the  Results 
Act  does  not  require  that  limitations  of  external  sources  of  data  be 
discussed  in  performance  plans,  we  believe  nasa’s  performance  plan 
would  be  improved  with  the  inclusion  of  such  a  discussion. 


Verifying  and  Validating  nasa’s  performance  plan  states  that  performance  will  be  evaluated  by 

Performance  internal  and  external  processes.  Internal  assessments  will  be  conducted  by 

agency  management  councils  throughout  the  year  and  semi-annually  by 
the  Senior  Management  Council.  External  assessments  will  be  conducted 
annually  by  the  nasa  Advisory  Coimcil  and  organizations  such  as  the 
National  Academy  of  Sciences.  For  example,  the  plan  indicates  that  the 
Aeronautics  and  Space  Transportation  Technology  Committee  of  the  nasa 
Advisory  Council  will  conduct  annual  assessments  of  the  progress  made 
by  the  Aeronautics  and  Space  Transportation  Technology  Enterprise.  This 
committee  will  provide  a  qualitative  progress  measurement,  including 
commentary  to  clarify  and  supplement  the  qualitative  measures.  Such 
commentary  would  constitute  a  means  to  verify  and  validate  measured 
values. 


Recognizing  Data  The  plan  falls  short  in  this  area,  nasa’s  plan  does  not  address  limitations  to 

Limitations  from  internal  or  external  sources.  To  gauge  its  progress,  NASA 

enterprises  wiU  likely  rely  extensively  on  internal  sowces  of  data,  such  as 
program  management  reports.  However,  the  plan  does  not  discuss  any 
limitations  that  these  data  sources  have.  For  example: 

•  As  previously  discussed,  the  performance  target  of  “improving  information 
technology”  wiU  require  assessments  of  the  trend  in  the  ratio  of  benefits 
achieved  to  information  technology  costs.  The  plan  would  be  more 
complete  if  it  identified  the  challenges  associated  with  determining 
benefits  attributable  to  information  technology  investments. 

•  In  describing  its  internal  crosscutting  process  characterized  as 
“Communicate  Knowledge,”  one  of  the  objectives  described  is  the 
development  of  educational  outreach  programs.  One  of  the  associated 
performance  targets  in  fiscal  year  1999  is  to  increase  the  munber  of 
students  reached  through  its  NEWEST/NEWMAST  programs  to  42,000 
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students  from  37,000  students.  It  is  not  clear  whether  NASA  will  do  its  own 
survey  or  need  to  rely  on  other  external  information  sources. 

NASA’s  performance  plan  could  also  benefit  from  a  discussion  of  the 
accuracy  and  rehabUity  of  data  from  external  sources,  even  though  such  a 
discussion  is  not  required  by  the  Results  Act.  As  previously  discussed,  the 
plan  recognizes  that  securing  collaboration  and  cooperation  with  other 
agencies  and  research  partners  is  key  to  the  successful  accompUshment  of 
its  goals.  In  its  description  of  the  objective  of  making  major  scientific 
contributions  to  national  and  international  environmental  assessments, 
the  plan  can  be  improved  by  indicating  whether  NASA  will  rely  on 
information  provided  by  its  partner,  the  Federal  Aviation  Administration 
(faa),  to  assess  atmospheric  effects  of  aviation,  and  if  so,  how  the 
accuracy  and  rehability  of  faa’s  data  wiU  be  assessed.  Since  the 
performance  target  associated  with  this  objective  is  to  make  significant 
contributions — further  defined  by  nasa  officials  as  being  referenced  in 
technical  literature — such  added  information  could  influence  greater  use 
of  the  information  generated. 

As  previously  noted,  a  critical  component  to  nasa’s  evaluation  of  its 
performance  is  the  establishment  of  a  financial  management  system  and 
its  integration  with  full  cost  accoimting.  Full  cost  accounting  facilitates 
decision  making  and  increases  accoimtability  by  automating  the  agency’s 
operations  and  linking  data  results  and  costs  to  mgjor  agency  activities, 
budgets,  accounts,  and  reports.  Unfortunately,  all  the  information  NASA 
needs  to  measure  the  costs  associated  with  its  performance  is  currently 
not  available,  nasa’s  plans  to  implement  full  cost  accounting  are 
contingent  upon  the  timely  completion  of  its  new  financial  management 
system,  which  NASA  plans  to  implement  in  1999.  Until  the  new  integrated 
financial  management  system  is  operational,  performance  assessments 
relying  on  cost  data  may  be  incomplete. 


Agency  Comments 
and  Our  Evaluation 


We  provided  a  draft  of  this  report  to  NASA  for  review  and  comment.  In  its 
comments,  NASA  stated  that  it  will  continually  improve  the  content  of  its 
annual  plan  and  recognizes  the  usefulness  of  our  comments  and 
recommendations,  nasa  mtends  to  use  its  strategic  plan,  budget 
justification,  and  performance  plan  as  a  “trilogy  of  documents”  to  be 
reviewed  as  a  package.  Even  though  the  areas  we  identified  in  our  report 
as  lacking  specificity  may  be  addressed  in  nasa’s  strategic  plan,  budget 
justification  and  other  documents,  we  maintain  that  the  performance  plan 
could  be  more  useful  by  better  linking  performance  goals  and  measures  to 
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program  activities  in  the  budget;  more  fully  explaining  procedures  for 
verifying  and  validating  performance  data  and  recognizing  the  limitations 
that  affect  the  credibility  of  data  that  VTill  be  used  to  measure 
performance;  and  by  acknowledging  major  management  challenges  and 
associated  corrective  actions.  Including  these  areas  in  the  plan  would 
provide  a  more  comprehensive  imderstanding  of  the  importance  of  NASA’s 
goals  and  performance  measures,  making  it  more  useful  for  purposes  of 
the  Results  Act. 

NASA’s  comments  and  our  evaluation  of  them  are  presented  in  appendix  I. 


Scope  and 
Methodology 


'The  scope  of  our  work  was  limited  to  the  information  contained  in  nasa’s 
fiscal  year  1999  performance  plan  and  clarifying  discussions  we  had  with 
NASA  officials.  The  criteria  we  used  to  conduct  this  review  were  the  Results 
Act,  omb’s  guidance  on  developing  the  plans  (Circular  A-11,  part  2),  our 
February  1998  guidance  for  congressional  review  of  the  plans  (ggd/AIMD 
10.1.18),  and  the  December  17, 1997,  letter  to  omb  Director  Raines  from 
several  congressional  leaders.  For  purposes  of  our  analysis,  we  collapsed 
the  six  requirements  for  armual  performance  plans  in  the  Results  Act  and 
the  related  guidance  into  three  core  questions:  (1)  To  what  extent  does  the 
agency’s  performance  plan  provide  a  clear  picture  of  intended 
performance  across  the  agency?  (2)  How  well  does  the  performance  plan 
discuss  the  strategies  and  resources  the  agency  will  use  to  achieve  its 
performance  plan?  (3)  To  what  extent  does  the  agency’s  performance  plan 
provide  confidence  that  its  performance  information  will  be  credible? 

We  conducted  our  work  between  December  1997  and  April  1998  in 
accordance  with  generally  accepted  government  auditing  standards. 


As  requested,  we  plan  no  further  distribution  of  this  report  until  7  days 
after  its  issue  date.  At  that  time  we  will  send  copies  to  the  Director  of  the 
Office  of  Management  and  Budget,  the  Administrator  of  NASA,  and 
appropriate  congressional  committees.  We  will  also  make  copies  available 
to  other  interested  parties  on  request. 
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If  you  or  your  staff  have  any  questions  concerning  this  report,  please 
contact  me  on  (202)  512-4841.  The  major  contributors  to  this  letter  are 
listed  in  appendix  II. 


Allen  Li 

Associate  Director 
Defense  Acquisitions  Issues 
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and  Space  Administration 


National  Aeronautics  and 
Space  Administration 

Office  of  the  Administrator 

Washington,  DC  20546-0001 


2  4  i^bci 


Mr.  Allen  Li 

Associate  Director 

Defense  Acquisition  Issues 

National  Security  and  International  Affairs  Division 

United  States  General  Accounting  Office 

Washington,  DC  20548 


Dear  Mr.  Li: 

Thank  you  for  the  opportunity  to  comment  on  GAO’s  draft  report  to  evaluate  the  NASA 
Performance  Plan  for  Fiscal  Year  1999.  We  are  pleased  that  you  have  concluded  that  the  Plan 
addresses  the  requirements  of  GPRA.  Since  we  intend  to  continually  improve  the  content  of  this 
annual  plan,  GAO’s  comments  and  recommendations  will  be  extremely  useful 

NASA  has  developed  the  Strategic  Plan,  Performance  Plan,  and  Budget  Justification  as  a 
‘'trilogy  of  documents”  to  be  reviewed  as  a  package.  Our  goal  is  to  enhance  the  usefulness  of 
these  plans  as  a  management  tool  by  limiting  duplication,  and  providing  concise  and 
comprehensive  information.  Many  of  the  areas  identified  in  the  report  as  lacking  specificity  are 
addressed  in  detail  in  the  Strategic  Plan  and/or  Budget  Justification.  This  includes  cooperation 
with  other  agencies,  detailed  program  strategies,  resource  requirements,  and  external  factors. 

Enclosed  is  our  response  to  the  issues  raised  in  the  GAO  report.  Due  to  time  constraints, 
these  comments  address  only  factual  errors  and  major  substantive  issues.  NASA’s  review  of  the 
subject  report  required  an  approximate  aggregate  of  60  staff  hours.  If  you  have  any  questions 
regarding  NASA’s  comments  or  would  like  to  discuss  them  further,  please  contact  Alan  M. 
Ladwig,  Associate  Administrator  for  Policy  and  Plans,  on  202/358-2096. 

Sincerely, 


^R/Dailey 

Deputy  Administrator 

Enclosure 
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NASA  Response  to  GAO’s 
“Observations  on  NASA’s  Performance  Plan 
Under  the  Results  Act” 

April  1998 


General  Observation 

The  report  offers  a  number  of  recommendations  to  improve  the  utility  of  the  Plan.  The  report 
frequently  places  emphasis  on  recommendations  for  additional  assessment  elements  which  we 
believe  are  beyond  the  scope  of  0MB  guidance  or  statutory  requirements.  Given  the  still 
embtyonic  stage  of  tlie  GPRA  reporting  process,  we  do  not  believe  it  is  constructive  to  add  new 
requirements  at  this  late  stage  in  the  preparation  cycle  of  the  Performance  Plan.  It  would, 
however,  be  useful  to  identify  specific  Sidelines  and  expectations  at  the  beginning  of  the 
planning  cycle. 

Measures  for  major  management  challenges 

On  pages  4,  5,  9, 1 0,  and  1 1,  the  GAO  report  indicates  that  NASA  should  include  management 
performance  targets  in  the  areas  of  contract  management,  information  systems,  and  financial 
management. 

Under  the  GPRA,  the  Performance  Plan  is  required  to  “address  annual  perfonnance  for  each 
program  activity.”  As  GAO  has  indicated  in  its  report,  NASA  has  clearly  met  this  requirement. 
To  demonstrate  the  Agency’s  recognition  of  the  importance  of  improving  our  management 
effectiveness  and  efficiency,  we  have  identified  goals,  objectives,  and  performance  measures  for 
our  Crosscutting  Processes, 

Although  not  required  by  GPRA,  NASA  has  included  top-level  indicators  of  management 
effectiveness  and  efficiency  to  support  internal  management  re\^ews.  However,  not  included  in 
the  Plan  are  detailed  measures  that  address  our  efforts  to  further  improve  information  systems 
technologies  and  capabilities,  systems  and  information  for  monitoring  contractor  acti\ities 
complying  with  contract  management  requirements,  and  implementation  of  a  new  accounting 
system.  The  goals,  objectives,  strategies,  and  performance  measures  for  these  management 
initiatives  are  reported  in  the  Chief  Financial  Officer’s  Five-Year  Plan,  the  Chief  Information 
Officer’s  Information  Technology  Implementation  Plan,  implementation  planning  documents 
being  developed  by  the  Office  of  Procurement,  as  well  as  Program  Management  Council 
activities. 

Resource  requirements  to  achieve  performance  targets 

On  page  9,  the  GAO  report  states  “Characterization  of  resources  needed  by  NASA  to  meet  its 
perfonnance  goals  is  limited  to  identifying  1999  funding  requirements  by  Enterprise.  In  addition, 
the  plan  does  not  discuss  the  operational  processes,  skills  and  technology,  human  and  capital 
resources  required  to  meet  the  performance  goals.” 

As  addressed  above,  NASA  has  not  included  detailed  program  resource  requirements  in  the 
Performance  Plan  since  this  information  is  already  provided  in  the  Budget  Justification.  NASA  is 
currently  evaluating  the  potential  for  integrating  the  Performance  Plan  and  Budget  Justification 
development  processes.  This  will  address  GAO’s  recommendation  to  include  a  crosswalk  matrix 
indicating  the  relationship  of  performance  targets  to  program  activities  and  resources  in  the 
budget.  We  understand  that  several  agencies  have  already  merged  these  two  documents.  NASA 
will  consult  with  OMB,  Congress,  and  GAO  for  guidance  regarding  the  integration  of  these 
documents  during  the  next  budget  cycle. 
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Implementing  Strategies 

On  pages  8  and  9,  the  GAO  report  states  that  "application  of  the  faster,  better,  cheaper  approach  to 
spacecraft  development  is  not  clearly  ejylained.”  It  further  expresses  a  desire  for  an  explicit 
discussion  on  what  is  involved  in  applying  the  approach  and  how  it  relates  to  prudent  risks. 

The  NASA  Perfonnance  Plan  clearly  states  that  "Pro^am  mailers  are  encouraged  to  accept 
prudent  risk,  shorten  development  time  of  technologies  and  missions,  explore  new  conceptud 
approaches,  streamline  management,  and  incorporate  innovative  methods  and  technologies  to 
enhance  efficiency  and  effectiveness.’*  NASA  management  bdieves  that  the  list  of  specific 
implementation  guidelines  actually  does  provide  a  clear  statement  of  the  application  of  the  "faster, 
better,  cheaper"  approach,  at  least  to  the  level  of  detail  appropriate  to  this  management-ie\'el 
document. 

Addressing  EOSDIS  in  the  Performance  Plan 

On  page  1 1 ,  GAO  indicates  that  EOSDIS  is  not  accounted  for  in  the  Plan. 

NASA  believes  this  observation  is  incorrect,  inasmuch  as  there  are  specific  perfonnance  metrics 
for  the  system  contained  on  page  18  of  the  Perfonnance  Plan.  NASA  considers  EOSDIS  to  be  a 
core  programmatic  draaent  of  the  Earth  Science  Enterprise.  Investments  made  in  the  system  are 
properly  accounted  for  from  that  standpoint,  rather  than  as  an  Agency  crosscutting  capital 
investment,  akin  to  the  Integrate  Financial  Management  Program  or  supercomputing.  These  latter 
investments  are  intended  to  enhance  productivity,  reduce  cost,  or  enable  new  capabiliti  es  across 
the  Agency,  something  that  EOSDIS  is  not  intended  to  accomplish 

Credible  Performance  Information 

On  pages  1 1  to  13,  the  GAO  report  raises  concerns  about  NASA’s  abiii^  to  assure  that 
performance  informatics  will  be  credible.  We  believe  that  it  is  premature  for  GAO  to  pass 
judgment  on  this  factor.  The  Agency  has  already  demonstrated  its  ability  to  accomplish  this  task 
during  the  devdopment  of  our  FY  1 996  and  FY  1 997  Accountability  Reports.  These  reports, 
which  include  a  performance  evaluation  section,  were  audited  by  Arthur  Andersen  LLP,  The 
audit  found  no  significant  issues  with  the  auditabiliy  and  credibility  of  NASA’s  performance 
information  plans. 
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GAO  Comments 


The  following  are  GAO’s  comments  on  NASA’s  letter  dated  April  24, 1998. 


1.  Responding  to  the  issues  raised  in  this  report,  NASA  concludes  that  it  has 
complied  with  the  requirements  of  GPRA.  We  agree.  Our  point  in  the 
report  was  that  the  plan  contains  areas  that  could  be  strengthened,  making 
it  more  useful  to  the  Congress.  In  discussing  the  strengths  of  NASA’s 
performance  plan,  our  report  recognizes  that  the  plan  provided  good 
linkage  between  strategic  goals  and  the  goals  in  the  performance  plan  and 
that  performance  measures  were  generally  objective,  quantifiable,  and 
useful  for  assessing  progress.  However,  our  report  also  clearly  states  that 
NASA’s  plan  could  provide  a  clearer  picture  of  intended  performance  across 
the  agency  and  does  not  fully  portray  how  nasa’s  strategies  and  resources 
will  help  it  achieve  the  plan’s  performance  goals.  The  report  also  states 
that  NASA’s  plan  does  not  clearly  explain  how  program  activities  relate  to 
the  budget;  and  that  there  is  no  crosswalk  to  facilitate  an  imderstanding  of 
the  relationship  between  the  specific  program  activities  in  the  budget  and 
the  goals  and  measures  in  the  performance  plan.  As  stated  in  our  report, 
NASA  is  in  agreement  to  merge  the  budget  justification  and  performance 
plan  process  in  an  effort  to  provide  a  crosswalk  showing  the  relationship 
of  performance  targets  to  program  activities  and  resomces. 

2.  In  commenting  on  measmres  for  major  management  challenges,  NASA 
recognizes  that  the  plan  does  not  include  measures  to  improve  the 
management  challenges  we  address.  While  these  management  issues  may 
be  addressed  in  other  management  dociunents,  such  as  the  Chief  Financial 
Officer’s  Five-Year  Plan,  Chief  Information  Officer’s  Information 
Technology  Implementation  Plan  and  implementation  planning  documents 
being  developed  by  the  Office  of  Procmement,  to  which  NASA  alludes,  we 
did  not  review  these  documents;  neither  does  the  performance  plan 
include  reference  to  them.  Referencing  these  documents  in  the  plan  would 
have  been  useful  in  providing  specific  sources  for  obtaining  a  more 
comprehensive  understanding  of  the  importance  of  these  issues.  We 
continue  to  believe  that  an  acknowledgement  of  major  management 
challenges  and  associated  corrective  actions  would  make  the  plan  more 
useful  for  purposes  of  the  Results  Act. 

3.  In  addressing  the  issue  of  resource  requirements  to  achieve 
performance,  nasa  correctly  restates  our  concern  that  the  plan  only 
addresses  funding  requirements  for  fiscal  year  1999  at  the  Enterprise  level. 
As  in  their  comment  to  our  point  on  not  identifying  management 
challenges,  nasa  makes  the  assumption  that,  since  a  discussion  of 
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operational  processes,  skills  and  technology,  human  and  capital  resources 
required  to  meet  the  performance  goals  is  provided  in  the  budget 
justification,  it  need  not  be  included  in  the  performance  plan.  Referencing 
these  documents  in  the  plan  would  have  been  useful  in  providing  specific 
sources  for  obtaining  a  more  comprehensive  understanding  of  the 
importance  of  these  issues. 

4.  Responding  to  our  concern  that  the  plan’s  implementation  strategy  of 
applying  the  “faster,  better,  cheaper”  approach  to  spacecraft  development 
is  not  clearly  explained,  nasa  contends  that  the  statement  that  follows  in 
the  plan  is  a  clear  enough  description  of  its  approach.  We  do  not  agree. 

The  statement  NASA  alludes  to  says  that  “Program  mant^ers  are 
encom^ged  to  accept  prudent  risk,  shorten  development  time  of 
technologies  and  missions,  explore  new  conceptual  approaches, 
streamline  management,  and  incorporate  innovative  methods  and 
technologies  to  enhance  efficiency  and  effectiveness.”  In  our  opinion,  this 
statement  does  not  provide  a  clear  explanation  of  nasa’s  implementation 
strategy.  We  continue  to  believe  that  a  more  explicit  discussion  of  this 
approach  is  needed. 

5.  Our  issue  regarding  eosdis  is  not  in  its  lack  of  identification  in  the 
section  discussing  crosscutting  information  technology.  Rather,  we  take 
issue  with  eosdis  not  being  specifically  mentioned  either  in  the  references 
cited  in  nasa’s  comments  or  anywhere  else  in  the  performance  plan, 
despite  the  fact  that  nasa  plans  to  spend  well  over  $1  billion  on  the  system. 
NASA’s  comment  assumes  that  the  reader  of  the  plan  would  recognize 
eosdis  as  the  vehicle  for  improving  dissemination  of  Earth  Science 
research  results.  We  believe  that  a  large  investment  such  as  eosdis  should 
be  explicitly  addressed  in  the  performance  plan  in  order  to  ensure  that  its 
actual  performance  is  understood.  For  example,  a  test  of  one  part  of  eosdis 
in  March  1998  revealed  software  problems  that  will  force  the  launch  of  the 
EOS  AM-1  satellite  to  be  delayed  by  at  least  6  months.  We  believe  that  the 
cost  and  schedule  impact  of  this  and  other  eosdis  development  problems 
needs  to  be  explicitly  and  objectively  assessed. 

6.  Responding  to  our  concern  about  the  lack  of  any  discussion  in  the  plan 
about  the  credibility  of  performance  information,  nasa  officials  expressed 
the  belief  that  the  data  it  collects  will  be  credible  and  that  we  may  be 
premature  in  passing  judgement.  As  support,  nasa  references  the  audit  of 
its  fiscal  year  1996  and  fiscal  year  1997  Accoimtability  Reports  by  Arthur 
Andersen  LLP,  which  the  agency  says  found  no  significant  issues  with  the 
credibility  of  nasa’s  performance  information  plans.  We  have  three  points 
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regarding  NASA’s  comment.  First,  the  plan  would  have  been  strengthened 
by  referencing  the  results  of  the  Arthur  Andersen  LLP  audit  in  the 
performance  plan.  Second,  while  a  review  of  the  Accountability  Reports 
was  not  in  the  scope  of  our  review,  we  believe  that  the  objectives  and 
performance  measures  in  the  performance  plan  are  likely  to  be  more 
results  oriented  than  those  included  in  the  Accountability  Reports  nasa 
references.  Third,  while  the  plan  does  identify  internal  and  external 
organizations  that  will  evaluate  its  performance,  we  believe  that  it  is  not 
premature  for  NASA’s  performance  plan  to  include  a  discussion  of 
procedures  that  will  be  used  to  verify  and  validate  performance  data  and 
the  limitations  that  could  exist  with  internal  sources  of  data  that  NASA 
plans  to  use  to  assess  performance. 
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NASA  Budget:  Carryover  Balances  in  Selected  Programs  (gao/nsiad-96-206, 
July  16, 1996). 

High-Risk  Program:  Information  on  Selected  High-Risk  Areas  (gao/hr-97-30, 
May  16, 1997). 

Results  Act:  Observations  on  NASA’s  May  1997  Draft  Strategic  Plan 
(GAO/NSIAD-97-205R,  July  22, 1997). 

NASA:  Major  Management  Problems  (GAO/'r-NSiAD-97-i78,  July  24, 1997). 

Managing  for  Results:  Agencies’  Annual  Performance  Plans  Can  Help 
Address  Strategic  Planning  Challenges  (gao/ggd-98-44,  Jan.  30, 1998). 


Aerospace  Testing:  Promise  of  Closer  nasa/DOD  Cooperation  Remains 
Largely  Unfulfilled  (gao/nsiad-98-52.  Mar.  11, 1998). 
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The  first  copy  of  each  GAO  report  and  testimony  is  free. 

Additional  copies  are  $2  each.  Orders  should  be  sent  to  the 
following  address,  accompanied  by  a  check  or  money  order 
made  out  to  the  Superintendent  of  Documents,  when 
necessary.  VISA  and  MasterCard  credit  cards  are  accepted,  also. 
Orders  for  100  or  more  copies  to  be  mailed  to  a  single  address 
are  discounted  25  percent. 

Orders  by  mail: 

U.S.  General  Accounting  Office 
P.O.  Box  37050 
Washington,  DC  20013 

or  visit: 

Room  1100 

700  4th  St.  NW  (corner  of  4th  and  G  Sts.  NW) 

U.S.  General  Accounting  Office 
Washington,  DC 

Orders  may  also  be  placed  by  calling  (202)  512-6000 

or  by  using  fax  number  (202)  512-6061,  or  TDD  (202)  512-2537. 

Each  day,  GAO  issues  a  list  of  newly  available  reports  and 
testimony.  To  receive  facsimile  copies  of  the  daily  list  or  any 
list  from  the  past  30  days,  please  call  (202)  512-6000  using  a 
touchtone  phone.  A  recorded  menu  will  provide  information  on 
how  to  obtain  these  lists. 

For  information  on  how  to  access  GAO  reports  on  the  INTERNET, 
send  an  e-mail  message  with  "info"  in  the  body  to: 

info@www.gao.gov 

or  visit  GAO’s  World  Wide  Web  Home  Page  at: 
http://www.gao.gov 
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